Methods and systems for processing multi-latticed video streams

ABSTRACT

An apparatus for facilitating reception of multiple representations of a video signal. In one embodiment, the apparatus includes a mechanism for receiving plural representations of the video signal corresponding to plural decimated versions of the video signal, associating pictures of the received plural representations of the video signal, and outputting pictures corresponding to information from associated pictures in accordance with a relative temporal order.

BACKGROUND

The present disclosure relates generally to data transfer in digitalnetworks and more specifically to improving error detection, correction,and/or concealment in digital video transmissions over digital networks.

Transfer of video stream over digital networks includes several aspects,such as video compression, error correction, and data-loss concealmentfor various types of communications networks and systems. Suchapplications often require robust systems and methods that facilitatedata transport with minimal data loss or perceived data loss. Systemsfor minimizing data loss or perceived data loss are particularlyimportant in applications such as video-broadcast applications usingpacket-switched networks, such as the Internet, where large burst errorsare common. Burst errors in packet-switched Internet Protocol (IP)networks may result from various mechanisms, including differences in IProuting times for different data packets transferred via the IP network.Unfortunately, conventional systems and methods for facilitating robustdata transfer with minimal data loss or perceived data loss often cannotaccommodate relatively large data losses without requiring excessivenetwork bandwidth and memory.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a communications system employing videopartitioning, time shifting, and Forward Error Correction (FEC)according to an example embodiment.

FIG. 2 is a diagram illustrating a first example partitioning of a videoframe by the system of FIG. 1.

FIG. 3 a is a first example timing diagram illustrating paralleltransmission of time-shifted video streams and an example data-lossinterval, where each stream corresponds to a partition or lattice of avideo signal processed by the example communications system of FIG. 1.

FIG. 3 b is a second example timing diagram illustrating exampletransmission timing of video data from a group of video framescorresponding to a video segment (t2) of FIG. 3 a.

FIG. 4 is diagram of the example video frame of FIG. 2 showing anexample data-loss pattern for the data-loss interval of FIG. 3 a for afirst video segment (t1).

FIG. 5 is diagram of a second example video frame partitioned inaccordance with the video partitioning illustrated in FIG. 2 and showingan example data-loss pattern for the data-loss interval of FIG. 3 a fora second video segment (t2).

FIG. 6 is diagram of a third example video frame partitioned inaccordance with the video partitioning illustrated in FIG. 2 and showingan example data loss pattern for the data-loss interval of FIG. 3 a fora third video segment (t3).

FIG. 7 is a diagram illustrating a second example partitioning of avideo frame by the system of FIG. 1.

FIG. 8 is diagram of the example video frame of FIG. 7 showing anexample data-loss pattern for the data-loss interval of FIG. 3 a for afirst video segment.

FIG. 9 is diagram of the example video frame partitioned in accordancewith the video partitioning illustrated in FIG. 7 and showing an exampledata-loss pattern for the data-loss interval of FIG. 3 a for a secondvideo segment.

FIG. 10 is diagram of the example video frame partitioned in accordancewith the video partitioning illustrated in FIG. 7 and showing an exampledata-loss pattern for the data-loss interval of FIG. 3 a for a thirdvideo segment.

FIG. 11 is a diagram illustrating a third example partitioning of avideo frame by the system of FIG. 1.

FIG. 12 is a third example timing diagram illustrating paralleltransmission of time-shifted video streams and an example data-lossinterval, where each stream corresponds to a partition or lattice of thevideo signal in the example communications system of FIG. 1.

FIG. 13 is diagram of an example video frame partitioned in accordancewith the video partitioning illustrated in FIG. 11 and showing anexample data-loss pattern for the data-loss interval of FIG. 12 for afirst video segment (t1).

FIG. 14 is diagram of the example video frame partitioned in accordancewith the video partitioning illustrated in FIG. 11 and showing exampledata-loss pattern for the data-loss interval of FIG. 12 for a secondvideo segment (t2).

FIG. 15 is diagram of the example video frame of FIG. 11 showing anexample data-loss pattern for the data-loss interval of FIG. 12 for athird video segment (t3).

FIG. 16 is a flow diagram of a first example method suitable for usewith the communications system of FIG. 1.

FIG. 17 is a flow diagram of a second example method suitable for usewith the communications system of FIG. 1.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

One embodiment of the invention maps a frame of a video signal with aplurality of matrices. For the purposes of the present discussion, amatrix may be any grouping of pixels or data associated therewith. Apixel may include one or more values associated with a data point, wherea data point may be a smallest displayable element or portion of a videoframe. A video frame may be any collection of data used to facilitateconstructing an image or representation thereof.

Each matrix may have a small number of pixels, n, such as, for example,where n=4, there are 4 pixels in a matrix. Note that in a specificembodiment n=p, where p represents the number of resulting streams, asdiscussed more fully below. Hence, a corresponding n number of streamsare formed, and the streams are transmitted over a network or channel ina time-skewed and/or time-interleaved manner, as discussed more fullybelow.

In a specific embodiment, an apparatus for processing and transmittingvisual information is disclosed. Visual information may be anyinformation from an information source such as from a camera, scannedfrom film, or synthetically created to form an image or portion thereof.The terms “visual information” and “image data” are employedinterchangeably herein. In a specific embodiment, the apparatus includesa first mechanism for mapping plural matrices onto a video frame. Amatrix is said to be mapped onto a video frame when a particular set orgroup of video data, such as video data corresponding to pixels, fromthe video frame is associated with or assigned to the matrix.

A second mechanism distributes n pixels in each matrix into ncorresponding decimated versions of the input video signal. A decimatedversion of a video signal may be any version of a video signal thatrepresents a subset of video data from the original or input videosignal. The decimated versions of the input video signal are also calledvideo partitions, latticed versions, lattices of the input video signal,or subsampled versions herein. Frames of a decimated version of an inputvideo signal are also called latticed frames, subsampled frames,decimated frames, or frame partitions. In general, the term “lattice”may describe a partition of a video signal, wherein each lattice of thevideo signal may include pixel information corresponding to a particularset of pixel locations in frames of a video signal. A particular set ofpixel locations is called a sampling region.

Note that video decimating, subsampling, partitioning, or latticing mayintroduce aliasing in the video signal, which may be removed via one ormore filter operations, such as anti-aliasing filters.

A third mechanism provides n streams from the n decimated video signalsaccording to a first relative temporal order. Each of the n decimatedvideo signals corresponds to a respective lattice, partition, decimatedversion, or subsampled version of the input video signal. For thepurposes of the present discussion, a stream may be any successivelytransmitted or received data, such as, but not limited to, compressedvideo frames of a video signal in a transmission order in accordancewith the syntax and semantics of a video compression specification, suchas Part 10 of the Moving Picture Experts Group (MPEG)—4 standard. Atemporal order may specify a temporal relationship between one or moresegments (e.g., a segment labeled t1, as discussed more fully below) ofa first version of a video signal occurring in a first stream (e.g., astream labeled S1, as discussed more fully below) and one or morecorresponding segments (e.g., a segment labeled t1) of a second versionof the video signal occurring in a second stream (e.g., a stream labeledS2, as discussed more fully below). Note that for the purposes of thepresent discussion, a video signal may be partitioned, decimated,subsampled, or latticed into multiple versions that may be processed orcompressed into multiple corresponding streams, and the multiple streamsmay be collectively called a digital video signal or a video signal.Furthermore, each of the multiple streams may also be called a videosignal.

A fourth mechanism implements instructions for changing the firstrelative temporal order to a second relative temporal order differentfrom the first relative temporal order. The second relative temporalorder is said to be a time-skewed and/or time-interleaved version of thefirst relative temporal order. For example, a first set of streamscharacterized by a first temporal order may include various segments ina first order or temporal relationship. A corresponding second set ofstreams with a second temporal order may include the various segments ofthe first set of streams, wherein the various segments are in adifferent temporal relationship, such that the order of the segments areskewed or separated by different time intervals than they are in thefirst set of streams.

An example method includes receiving a video signal with one or morevideo frames; partitioning each of the one or more video frames of thevideo signal into plural decimated video frames; and then separating orprocessing the respective sequences of decimated video frames into oneor more identifiable video streams. Each of the one or more identifiablevideo streams may be segmented into consecutive segments, wherein eachconsecutive segment comprises of one or more consecutive compressedvideo frames. In the present example method, each compressed video frameis associated with n decimated video frames, where n represents thenumber of pixels in a matrix, i.e., specific grouping of pixels, that ismapped contiguously over the non-decimated video frame. Segments of eachof the one or more identifiable video streams are strategically shiftedin time relative to corresponding segments of one or more otheridentifiable video streams. This time shifting or skewing occurs beforetransmission to facilitate error correction and/or error concealment, asdiscussed more fully below. Transmission of the one or more identifiablevideo streams results in time-shifted transport of multi-latticed video.

Example Embodiments

A more specific embodiment implements an apparatus for separating avideo signal into plural identifiable lattices, also callederror-concealment partitions, decimated versions, or subsampledversions. Such lattices are not to be confused with the layers of ascalable video coding method. Each of these latticesis associated with,or “assigned,” pixel data from a corresponding set of pixel locations,also referred to as a “sampling region,” in each frame of the videosignal. Each lattice includes pixel information from particulardecimated versions of video frames, and a given decimated version of avideo frame is included within a particular lattice of the video signal.Each set of sampled pixel locations providing a distinct decimated videoframe is said to form a lattice of pixels, also called a latticed frameor sub-frame, in the given video frame. The multiple latticed frames ofvideo frames of a given video signal determine respective sequences ofdecimated video frames that may be processed and transmitted as separatestreams. Each separate sequence of decimated video frames is called alattice of the original or input video signal and is compressed andsegmented into consecutive video segments, and Forward Error Correction(FEC) processing is applied thereto, as discussed more fully below. Eachseparate sequence of decimated video frames of the input video signal isa respective independent representation of the video signal. Thepictures of each respective representation may be processed orcompressed independently from other representations of the input videosignal. Thus, each respective representation of the video signal is anindependent representation of the video signal since it may be processedor compressed independently of the other remaining representations ofthe video signal. Each resulting sub-video stream (i.e., a decimatedversion of the video signal in compressed form), may be processed ordecompressed independently of the other sub-video streams.

Each segment of a stream or segment of a video signal can include one ormore consecutive video frames in their transmission order. Theconsecutive video frames may be compressed video frames. A compressedvideo frame may be any frame to which a compression algorithm or otheroperation has been applied to reduce the number of bits used torepresent the video frame. Each of the consecutive video frames in agiven video stream corresponds to a respective decimated video framederived from a frame of an original or input video signal. Video framesin different streams are said to correspond with each other if they arerespective decimated versions or representations derived from a samevideo frame in the original or input video signal.

Corresponding segments in separate streams, as described below, may beshifted in time relative to each other so that a data loss during agiven time interval will not corrupt all of the decimated video framesassociated with a given frame of the input video signal. Consequently,missing or corrupted portions of a frame, such as a compressed frame,may be concealed via various mechanisms, including linear or nonlinearinterpolation or frame upscaling, at the receiver as discussed morefully below. Hence, this embodiment combines error correction and errorconcealment to facilitate resilient robust transport of video over alossy channel or network, such as an Internet Protocol (IP)packet-switched network. Certain embodiments discussed herein may beparticularly useful in applications involving broadcasting video viapacket-switched networks, also called over-the-top video transmission.

Note that FEC techniques may be applied to a given data stream to betransmitted over a network. Application of FEC to a data stream involvesadding redundant data to the data stream to reduce or eliminate the needto retransmit data in the event of certain types of data loss. Theredundant data facilitates reconstructing the data stream at a receiverin the event of data loss. Data may be lost due to noise, differing IProuting convergence times, Raleigh fading in wireless networks, and soon. Application of FEC to a data stream may also include the correctionof lost data or other errors in a data stream using the redundant data.

Unfortunately, due to excessive overhead and bandwidth constraints ofmany communications systems, certain conventional FEC systems often donot correct or adequately compensate for large losses, such as burstcorrelated losses of more than 500 milliseconds. This may result, forexample, in undesirable blank frames in transported video andcorresponding black screens in the resulting displayed video signal.Such problems may be addressed by certain embodiments discussed herein.

For clarity, various well-known components, such as video amplifiers,network cards, routers, Internet Service Providers (ISPs), InternetProtocol SECurity (IPSEC) concentrators, Media GateWays (MGWs), filters,multiplexers or demultiplexers, transport streams, and so on, have beenomitted from the figures. However, those skilled in the art with accessto the present teachings will know which components to implement and howto implement them to meet the needs of a given application.

For the purposes of the present discussion, electronically transporteddata may be any data that is communicated from a first location to asecond location via electromagnetic energy. Examples of electronicallytransported data include data transferred over packet-switched networksvia Internet Protocol (IP), data transferred via circuit-switchednetworks, such as the Public Switched Telephone Network (PSTN), and datatransferred wirelessly using a wireless protocol, such as Code DivisionMultiple Access (CDMA), Advanced Mobile Phone Service (AMPS), WiFi(Wireless Fidelity), WiMAX (Worldwide Interoperability for MicrowaveAccess), and Bluetooth protocols.

FIG. 1 is a diagram of a communications system 10 employing videopartitioning, time shifting, and Forward Error Correction (FEC)according to an example embodiment. The communications system 10includes a transmitter 12 in communication with a first receiver 14 anda second receiver 30. For the purposes of the present discussion, acommunications system may be any device or collection of devices thatcontains one or more components that intercommunicate or are otherwiseadapted to intercommunicate.

The transmitter 12 includes a video-partitioning module 16, which iscoupled to a video time-shifting module 18. The video time-shiftingmodule 18 is coupled to a video encoder 20. Although the videotime-shifting module 18 is shown prior to video-compression module 22,other embodiments may perform a time-shifting function at differentpoints in a processing path. For example, an alternate embodiment mayhave the video time-shifting module 18 at the output ofvideo-compression module 22. Other arrangements are possible. The videoencoder 20 includes a video-compression module 22, which is coupled to atransmit chain 24. The transmit chain 24 includes an FEC module 26. Inone embodiment, the transmitter 12 includes filtering capabilities inthe video-partitioning module 16. Such filtering capabilities mayinclude linear, non-linear, or anti-aliasing filtering capabilities.

A first decimated video frame is said to correspond to a seconddecimated video frame if they both originated from the same frame of theinput video signal. That is, the video-partitioning module 16 producedthe first and second decimated frames from the same frame of the inputvideo signal, and, thus both decimated frames correspond temporally tothe same instance or interval of time for display or output purposes.Likewise, a plurality of corresponding decimated video frames refers toa plurality of decimated video frames that originated from the sameframe of the input video signal and correspond temporally to the sameinstance or interval of time for display or output purposes.

A compressed video frame in a first video stream is said to correspondto a compressed video frame in a second video stream if both of thesecompressed video frames prior to being compressed were correspondingdecimated frames (i.e., both originated from the same frame of the inputvideo signal). Likewise, respective compressed video frames in aplurality of video streams are said to be corresponding compressed videoframes if each of them originated from the same frame of the input videosignal. Depending on the embodiment, partitioning of frames may occurbefore or during their compression. Depending on the embodiment,time-shifting of frames or video segments may occur before, during, orafter their compression. Depending on the embodiment, partitioning andtime-shifting may or may not occur in the same processing module oftransmitter 12.

For purposes of illustrating a particular embodiment, let frame (k, v)represent the k-th frame in transmission order of a given video stream vsuch that frame (k, 1) is the k-th video frame in a first videostream;frame (k, 2) is the corresponding k-th frame in a second videostream; and frame (k, p) is the corresponding k-th frame in the p-thvideo stream. For nf (number of frames) equal to a positive integer, asegment of nf consecutive frames of the first video stream is said tocorrespond temporally to a segment of nf consecutive frames of thesecond video stream if for each integer value of k from 1 to nf, therespective k-th frames in transmission order are corresponding frames.Similarly, a plurality of segments in respective video streams are saidto be temporal corresponding segments, or just corresponding segments,if all possible pairing of two of the plurality of segments are temporalcorresponding segments. That is, p video segments are said to becorresponding segments if: (1) each of the p segments has the samenumber of frames, nf, and (2) in transmission order, for each integervalue of k from 1 to nf, the kth frame in each of the p segments is acorresponding frame to the respective kth frames in the other (p−1)segments.

Video compression module 22 outputs the successive compressed videoframes in each of the p video streams in accordance with the syntax andsemantics of a video coding specification. Encoder 20 may specify use ofa transport stream into which multiple streams of a video signal aremultiplexed via the transmit chain 24 before transmission over thenetwork 32. The transmission order of the successive compressed video ina video stream may or may not equal the display or output order of theframes. For example, in certain applications, a future reference framemay be required to be transmitted prior to a frame having an earlierdisplay or output time, but that depends on the decoded version of thatfuture reference frame for its reconstruction. The video compressionmodule 22 effects compression of the p decimated video signals such thatthe relative transmission order of the successive compressed videoframes in each of the corresponding p video streams is the same.However, in the present embodiment, although the relative transmissionorder of the frames within each of the p video streams is the same, asexplained below, each set of p corresponding video segments istransmitted in accordance with a second relative temporal order, whichis a skewed or time-shifted version of a first relative temporal order.

The transmitter 12 is coupled to the first receiver 14 and the secondreceiver 30 via a network 32. The network 32 may be the Internet, awireless network, or other type of network or communications channel(s).Although multiple receivers or decoders are described herein, otherembodiments may use a single decoder or receiver for one or more videostreams.

The first receiver 14 includes a first decoder 34, which is coupled to afirst video-de-partitioning module 42, which is coupled to a firstloss-concealment and latency-compensation module (LCALCM1) 44. The firstdecoder 34 includes a first receive chain 36, which includes a firstreverse-FEC module 38. The first receive chain 36 is coupled to a firstvideo-decompression module 40 in the first decoder 34. The first decoder34 is coupled to the first video-de-partitioning module 42, which iscoupled to the first loss-concealment and latency-compensation module44. The second receiver 30 is similar to the first receiver 14, with theexception that the second receiver 30 is adapted to subscribe to asubset of the video streams to which the first receiver 14 subscribes,and may include certain filtering capabilities not included in firstreceiver 14, such as frame upscaling capabilities, or additional ordifferent frame upscaling capabilities.

For the purposes of the present discussion, a video stream may be anysuccessively transmitted portions of video data, such as, but notlimited to, the compressed video frames of a video stream in atransmission order that is in accordance with the syntax and semanticsof a video compression specification. One example video stream includesa series of sequentially transmitted video packets. Transmission of avideo frame may require transmission of multiple video packets.

In one embodiment, plural separate video streams are multiplexed into asingle transport stream and then transmitted over a single transmissionchannel. Auxiliary information is provided in the transport stream toidentify the video streams. The auxiliary information may includeinformation indicating how decoded versions of compressed video framepartitions are to be assembled into a larger frame to be displayed oroutput. The auxiliary information may also include informationindicating the relative temporal order of latticed frames andcorresponding segments in the transport stream, as discussed more fullybelow.

In an alternative embodiment, a video stream includes plural sub-videostreams packetized and appropriately multiplexed within the videostream, such as by interspersing video packets carrying thecorresponding lattices in their compressed form in the video stream toeffect a parallel or simultaneous transmission of plural decimated videoframes, i.e., latticed frames, via the sub-video streams. In thisembodiment, each of the sub-video streams corresponds to a respectivepartition of the input video signal. Auxiliary information in the videostreams provides identification information that conveys spatialrelationships of the lattices and the relative temporal order of thecompressed video segments or compressed video frame partitions. For thepurposes of the present discussion, the relative temporal order of videosegments and/or frames (or sub-frames) in a video stream may specify theactual order of the start, end, or completion of each set ofcorresponding video segments and/or each corresponding frame in thevideo stream and may further specify the time intervals between thestart, end, or completion of each video segment in the video streamand/or of each frame in the video stream. When the sequence ofcompressed frames is transmitted over a channel, the relative order oftransmitted frames may be called the relative temporal transmissionorder, relative transmission order, or relative temporal order. Therelative temporal order of video segments or frames is said to berelative, since they are ordered or positioned for transmission withrespect to each other in intervals of time or in intervals ofconsecutive frames of the input video signal.

Plural video streams that are transmitted in parallel are notnecessarily transmitted simultaneously or multiplexed over a commontransmission channel. In another embodiment, plural video streamscorresponding respectively to video partitions are transmitted inparallel over two different transmission channels between a firstlocation and a second location with the appropriate synchronizationprovisions as well as the identification information. Such channels arecalled parallel channels.

The second receiver 30 includes a second receive chain 46, whichincludes a second reverse FEC module 48. A second decoder 56 includes asecond reverse FEC module 48 and a second video-decompression module 50.The second decoder 56 is coupled to a second video-de-partitioningmodule 52. The second video-de-partitioning module 52 is coupled to asecond video concealment and latency-compensation module (LCALC2) 54.

The loss-concealment and latency-compensation modules 44, 54 areerror-concealment modules. For the purposes of the present discussion,an error-concealment module may be any entity that is adapted todisguise an impairment in a video stream, such as omitted data, lostdata, impaired data, or data that has not yet been received by areceiver, or other errors occurring in the transmission or reception ofa video stream. Herein, an impairment refers to omitted data, lost data,impaired data, or data that has not yet been received by a receiver, orto other errors occurring in the transmission or reception of a videostream.

The LACLM1 44 and LACLM2 54 include filtering capabilities, such aslinear, non-linear or anti-aliasing filtering capabilities to effectupscaling of a decoded video frame. The filtering capabilities in theLACLM1 44 and LACLM2 54 may compensate for lost data, impaired data, ornon-received data. For example, filtering capabilities may be employedto upscale at least a portion of a decoded frame in a first video streamto conceal an impairment in a corresponding frame of a second videostream. For the purposes of the present discussion, data is said to beupscaled when deriving or replicating data to compensate for animpairment of data.

The filtering capabilities in the LACLM1 44 or LACLM2 54 may be employedto upscale at least a portion of a decoded version of frame (k, 1) thatspatially-corresponds to the impaired portion of frame (k, 2). Some orpossibly all of the generated pixel values in the upscaled version ofdecoded frame (k, 1) are used to compensate for the corresponding pixelsof at least one impaired portion of the decoded version of frame (k, 2)or the whole of frame (k, 2) if frame (k, 2) was completely impaired orundecodable.

In one embodiment, when frame (k, 2) is impaired, a single decodednon-impaired_frame, e.g., frame (k, 1), is upscaled in LACLM1 44 orLACLM2 54 to compensate for the respective one or morespatially-corresponding impaired portions in frame (k, 2). Alternativelyor in addition, when frame (k, 2) exhibits one or more partial-frameimpairments, one or more portions of a single decoded non-impairedframe, e.g., frame (k, 1), are upscaled in LACLM1 44 or LACLM2 54 tocompensate for the respective spatially-corresponding impaired portionsin frame (k, 2).

In another embodiment, p video streams and identification informationare received at the first receiver 14. Filtering capabilities in LACLM144 are used to upscale the decoded versions of (p−1) correspondingnon-impaired frames to compensate for one or more impairments in thecorresponding frame of the p-th video stream. For the purposes of thepresent discussion, the respective k-th frames in each of p receivedvideo streams are considered corresponding video frames. The relativetemporal order of the p corresponding k-th video frames is determined bythe first receiver 14 from the received identification information,i.e., auxiliary information, which may be included in a transportstream. The relative temporal order of the p corresponding k-th framesmay be determined by the first receiver 14 from the receivedidentification information that identifies the relative temporal orderof the p corresponding video segments. The corresponding p video framesare decoded by the first decoder 36 in accordance with their relativetemporal order. The spatial relationships of the decoded versions of thep corresponding k-th video frames are also determined by the firstreceiver 14 from the same received identification information.

In an alternate embodiment, the spatial relationships of the decodedversions of the p corresponding k-th video frames are determined by thefirst receiver 14 from additional or different auxiliary informationthat differs from the received identification information describedabove. A composite or assembled video frame of the same spatialresolution as the input video signal to the transmitter 12 is formed bythe first video de-partitioning module 42 in accordance with the spatialrelationships determined from the identification information or from theadditional or different auxiliary information. One or more of the (p−1)corresponding decoded frames are individually upscaled in the LACLM1 44to compensate for one or more impairments in the k-th frame of the p-thvideo stream. Similarly, when two corresponding frames in two differentreceived video streams exhibit impairments, one or more of the (p−2)corresponding decoded frames are individually upscaled to compensate forthe impairments. The second receiver 30 may operate similarly to thefirst receiver 14 with the exception that the second receiver 30receives or subscribes to less than p video streams, as discussed morefully below.

Note that various couplings between modules and groupings of modulesshown in FIG. 1 are for illustrative purposes. Those skilled in the artmay employ different couplings and groupings without departing from thescope of the present teachings. For example, the encoder 20 may includethe video-partitioning module 16 and the time-time shifting module 18.In addition, the FEC module 26 may be included before and/or after thevideo compression module 22 and may also be considered part of theencoder 20. The exact couplings and order of various modules of FIG. 1are application specific and may be readily changed to meet the needs ofa given application by those skilled in the art without undueexperimentation.

In one embodiment, the first receiver 14 and the second receiver 30 arehoused within one physical receiver capable of receiving and processingtwo video transport streams, where each transport stream includesmultiple sub-video streams, simultaneously. In this embodiment, one ormore of the similar or same functional modules in the first receiver 14and the second receiver 30 may be performed by the same or commonphysical processing device.

In operation, the video-partitioning module 16 receives a video signalas input. In the present embodiment, the input video signal is adigitized and uncompressed video signal that is ingested as a sequenceof digitized pictures, or video frames, in their temporal display oroutput order and in accordance with a digital video or video interfacespecification. The digital video or video interface specification mayspecify use of a pixel clock, a picture format, a number of pictures persecond, a pixel format, and/or the scan or serial order of the pixels ofvideo frames, or other attributes and values. The scan format of theinput video may correspond to a progressive or interlaced video signal.The resulting ingested video signal is said to include or representvideo data. The exact picture format, number of pictures per second,pixel format, and scan format of the received video data may beapplication specific. Different types of video formats may be used fordifferent applications.

For the purposes of the present discussion, video data may be any datacomprising a video signal. A video signal may be any sequence of one ormore pictures or image data that can be displayed. Image data may be anyinformation born from a camera, scanned from film, or syntheticallycreated to form an image or portion thereof. An image may be a visualrepresentation of anything. A video frame may be any collection of imagedata used to facilitate constructing an image or representation thereof.The collection of data may include plural pixels of information, where apixel may include one or more values associated with a data point, wherea data point may be a smallest displayable element or portion of a videoframe. The terms “picture”, “frame,”, “video frame,” and “image frame”are employed interchangeably herein

A video-partitioning module, such as the module 16, may be any hardwareand/or software device, collection of devices, or other entity that isadapted to identify, separate, or mark different lattices, i.e.,partitions, of a video signal. The video-partitioning module 16 includescircuitry and instructions, which may include one or more softwareand/or hardware routines for selectively latticing the input frames ofthe input video signal, thereby separating the input frames of the inputvideo signal into different decimated frames, called latticed frames. Inthe specific embodiment of FIG. 1, the video-partitioning module 16samples each input frame to obtain smaller latticed frames. The latticedframes include pixel information from particular sampling regions, whichrepresent sets of predetermined spatial locations of pixels, where thepixels are selected from matrices of each input frame.

Pixels of each frame are grouped into matrices. In one exampleembodiment, the matrices of a frame include 4×4 non-overlapping groupsof adjacent or contiguous pixels. The assignment of pixels of a frame tomatrices may be considered a two-dimensional mapping of non-overlappingcontiguous matrices on a frame.

In another example embodiment, each frame of video data in a videosignal is separated into p (e.g., four) sampling regions or partitions,where each of the p sampling regions corresponds to one of p differentdecimated video signals, i.e., lattices, output by thevideo-partitioning module 16. Each of the p partitions, i.e., lattices,corresponds to a set of pixel locations representative of a samplingregion. The actual pixels (or values associated with the pixels) at thepixel locations of particular sampling regions of input frames representpixels of a lattice. The pixels of a sampling region for a single framerepresent a latticed frame. A lattice may include multiple latticedframes, which together represent a partition of a video signal. In thepresent specific embodiment, partitions or lattices of a video signalare conveyed via streams, called sub-video streams or sub-streams. Thesub-streams may be transmitted across the network 32 in a one or morecollective streams, such as a single transport stream, depending on theparticulars of a given implementation.

The pixels selected for each of the p lattices are dispersed across aframe in accordance with the mapping of the non-overlapping contiguousn-pixels matrices on the frame. For example, in one operational modewhere the number (n) of pixels in each matrix is four (n=4) and thenumber (p) of partitions or streams (also called sub-video streams orsub-streams) formed from the input video signal is four (p=4), a videoframe with a picture resolution of 640 pixels in the horizontal and 480pixels in the vertical is mapped with a 320 by 240 grid of 2×2 matrices,and thus, the video frame is divided into different groups (matrices) offour pixels. Each 2×2 matrix contains four “adjacent” or neighboringpixels per the meaning of adjacency described below. Each pixel in a 2×2matrix is allocated to one of the four lattices, i.e., partitions, whichare each conveyed via one of the four streams. Note that a video framemay be mapped with matrices of different sizes and shapes other than 2×2pixel matrices without departing from the scope of the presentteachings.

A pixel is said to be spatially adjacent, or adjacent, to another pixelif they are positioned directly next to each other, either horizontallyor vertically. In an alternate embodiment, pixels may be also consideredadjacent if diagonally next to each other. For example, two pixels maybe considered adjacent if at least one corner of a first pixel isadjacent to at least one corner of a second pixel.

Each matrix in the mapped two-dimensional grid of non-overlappingcontiguous matrices on an input video frame corresponds to a samplingregion, where the sampling region represents the locations of the pixelsof the matrix. The shape of a sampling region corresponding to a mappedmatrix may be square, rectangular, linear, or polygonal. In the presentspecific embodiment, the sampling regions have horizontal and verticaledges as defined relative to edges of a frame. For example, if arectangular frame is tilted, the edges of sampling regions within thetilted frame are still considered to be horizontal or vertical.

Two adjacent mapped matrices separate adjacent pixels located acrosstheir horizontal or vertical edges. In one embodiment, each mappedmatrix in a video frame is adjacent to at least one other mapped matrix.Alternatively, each mapped matrix in a video frame is adjacent to atleast two other different mapped matrices. Alternatively, each mappedmatrix in a video frame is horizontally adjacent to at least one othermapped matrix and vertically adjacent to at least one other mappedmatrix. Alternatively, each mapped interior matrix in a video frame isadjacent to at least four other different mapped matrices. The bordersof an interior matrix do not coincide with or are not adjacent to anyportion of a video frame's borders.

In one embodiment, all of the mapped matrices onto a frame have the sameshape and size. In an alternative embodiment, alternating mappedmatrices in scan order differ in size. In another embodiment, thealternating mapped matrices in scan order differ in shape. In yetanother embodiment, the alternating matrices in scan order differ inshape and size. Hence, successive mapped matrices in scan order maydiffer in shape and/or size without departing from the scope of thepresent teachings.

In one embodiment, the mapped matrices onto a frame do not overlap. Inan alternative embodiment, the mapped matrices onto a frame overlap.Hence, mapped matrices may or may not spatially overlap.

Each mapped matrix contains n pixels that are distributed by the videopartitioning module 16 into p partitions or lattices. In one embodiment,the number of pixels in a mapped matrix equals the number of partitions(i.e., n=p), and each partition has the same number of pixels. That is,each of the p distinct decimated versions of the input video signal hasthe same spatial frame resolution. In an alternative embodiment, p isless than n, and n/p is an integer, and the p partitions have the samespatial frame resolution. That is, the video-partitioning module 16 maydistribute (n/p) pixels from each mapped matrix into each of the ppartitions.

In yet another embodiment, p is less than n, and n divided by p does notequal an integer number, and at least one of the p partitions has aspatial frame resolution that is different from the respective spatialframe resolution of the other decimated video signals.

Note that in certain embodiments or implementations, thevideo-partitioning module 16 may include methods or instructions forselectively adjusting the subsampling patterns or mapped matricesemployed by the video-partitioning module 16 according to one or morepredetermined criteria. For example, the subsampling patterns may beselected so that any data loss is more easily concealed or disguisedbased on one or more characteristics of human perception. For example,humans may not be able to perceive an improvised reconstruction of lostpixel data occurring along a diagonal direction of pixels in a frame ordisplay screen as easily as they may be able to perceive lost pixel dataoccurring horizontally or vertically across a display screen.Accordingly, the subsampling patterns may be selected to force datalosses within a predetermined time interval to occur in patterns otherthan horizontal or vertical lines, as discussed more fully below.

In the present specific embodiment of FIG. 1, the video-partitioningmodule 16 outputs p separate decimated video signals derived from theinput video signal, which may be in the form of a sequence of digitizeduncompressed frames in the native frame display order of the input videosignal.

Output decimated video signals are provided to the time-shifting module18. The p separate decimated video signals are provided to thetime-shifting module 18 in parallel (i.e., at the same time). Thetime-shifting module 18 segments each of the p decimated video signalsinto consecutive video segments such that the start and end of each ofthe consecutive segments in consistent for each of the p video signalsto maintain temporal corresponding segments. The time-shifting module 18outputs the successive sets of p corresponding segments to the videocompression module 22 in accordance with a first relative temporalorder.

In one embodiment, the number of consecutive decimated frames in eachsegment of each successive set of p corresponding segments is fixed. Inan alternative embodiment, the number of consecutive frames, nf, in twoconsecutive video segments of a given decimated video signal changesfrom a first number to a second number. The change from a first numberof consecutive frames to a second number of consecutive frames alsooccurs for the corresponding segments of the other p−1 decimated videosignals.

Note that the p parallel time-shifted decimated video signals output bythe time-shifting module 18 may be created from a single input videosignal rather than from four parallel signals output by thevideo-partitioning module 16. For example, the video video-partitioningmodule 16 may alternatively output video data to a memory or storagedevice (instead of directly to the time-shifting module 18). Theresulting stored video data in the memory or storage device may includeidentification information identifying video lattices and segmentsthereof within the stored video data. The identification information mayfurther specify the number of decimated frames in video segments, thetemporal order of video segments in the respective p decimated videosignals, and may further specify the temporal relationships betweendifferent video segments.

The input video signal to transmitter 12 may include a sequence ofdigitized uncompressed frames, including video frames that are mapped,via the video-partitioning module 16, with non-overlapping contiguousmatrices containing n pixels each. For an embodiment in which p=n=4,each pixel of each mapped matrix is strategically assigned to adifferent one of the four parallel decimated video signals output by thevideo-partitioning module 16 and then processed by the time-shiftingmodule 18. Each of the four parallel partitions gets one pixel fromevery mapped matrix in a given frame. Values of each distributed pixelto a partition may be altered from the corresponding pixel values of theinput video signal by filtering capabilities in video-partitioningmodule 16.

Each decimated video signal is said to correspond to one or morepartitions or lattices of the input video signal. Each frame includessampling regions, the pixels of which are allocated to lattices, whichare also called partitions. In the present embodiment, the samplingregions comprise locations corresponding to alternating pixels in therows and columns of frames of the input video signal, which is input tothe video-partitioning module 16.

In an alternative embodiment, a given stream (also called sub-stream),i.e., a given one of the p video signals output by thevideo-partitioning module 16, may include plural lattices or partitionsof a given video frame. In this case, each frame of the input videosignal (input to the video-partitioning module 16) is latticed intolatticed frames, wherein plural latticed frames from a given input frameare allocated to a given stream. In this example, the number of streams(p) is less than the number of pixels (n) in a matrix.

In the embodiment where p=n=4 and where each frame is mapped with atwo-dimensional grid of non-overlapping contiguous 2×2 matrices, a firstdecimated video signal of the four decimated video signals (streams)output by the video-partitioning module 16 includes one or more pixelslocated in upper left portion(s) of the set(s) of pixel locationscorresponding to one or more mapped 2×2 matrices. A second decimatedvideo signal includes one or more pixels located in upper rightportion(s) of the set(s) of pixel locations corresponding to the mapped2×2 matrices. A third decimated video signal includes one or more pixelslocated in lower left portion(s) of the set(s) of pixel locationscorresponding to the mapped 2×2 matrices. A fourth decimated videosignal includes one or more pixels located in lower right portion(s) ofthe set(s) of pixel locations corresponding to mapped 2×2 matrices. Theparticular mapping of 2×2 matrices are selectively repeated across eachframe of the video signal so that each of the four decimated videosignals include a different set of pixels chosen from every other pixelon every other line of each video frame of the input video signal.

Note that more or fewer than four pixels and four different lattices maybe employed without departing from the scope of the present teachings.For example, the video-partitioning module 16 may partition the inputvideo signal into two (instead of four) decimated video signals(streams), which are output in parallel to the time-shifting module 18.

Alternatively, the video-partitioning module 16 can identify differentlattices of the input video signal and output a single uncompressedvideo signal to the time-shifting module 18 in response thereto. In thisalternative implementation, the single uncompressed video signal outputby the video-partitioning module 16 is complemented with auxiliaryinformation that identifies the different lattices or partitions of thesingle uncompressed video signal. Exact details for implementingpartitioning or latticing of an input video signal can be applicationspecific.

For example, in an alternative implementation, the video-partitioningmodule 16 provides auxiliary information identifying the differentlattices of the input video signal. The information identifying thedifferent lattices enables the time-shifting module 18 to selectivelychange the temporal relationship between different segments of p videoconstituents (corresponding to the streams of other implementationsdiscussed above) of the single stream output by the video-partitioningmodule 16. The p video constituents may be output by the time-shiftingmodule 18 as individual streams. The auxiliary information identifyingthe different lattices may also facilitate enabling the receiver 14 toreconstruct a video signal from plural decoded video streams that havebeen time-shifted with respect to each other in accordance with a secondrelative temporal order, which has been skewed relative to a firsttemporal order, as discussed more fully below. The identification ofdifferent lattices via auxiliary information may be implemented viavarious mechanisms, such as by insertion of specific identifyingpackets; by selectively adding or altering packet headers at thetransport stream level, the packetized elementary stream level, thecoded video layer; or by other mechanisms. Alternatively, identificationinformation is provided in data fields in: a transport stream's packetheader or outside a packet payload. In another embodiment, theidentification information is provided in data fields in a packetizedelementary stream's packet header or outside the packet payload, whereinthe packetized elementary stream is carried in the payloads of transportstream packets. In yet another embodiment, the identificationinformation is provided in data fields in a packet header or outside apacket payload of a coded video layer.

Alternatively, identification information conveys the temporalrelationship(s) between p non-corresponding video segments multiplexedand transmitted by the transmitter 12 over each successive transmissiontime interval, as discussed below with reference to FIG. 3 a, whereineach of the p non-corresponding video segments respectively correspondsto a different video stream.

Those skilled in the art with access to the present teachings mayreadily implement video partitioning and de-partitioning to meet theneeds of a given implementation without departing from the scope of thepresent teachings.

In one embodiment, the same sampling regions (used to obtain differentlattices or partitions) are used by the video-partitioning module 16 foreach successive frame of the input video signal. Each partitioncomprises pixels at the same spatial locations in every successive inputframe, and the sampling regions are said to be two-dimensional samplingregions. Note that a sampling region represents a set of pixellocations, where the pixels at the locations in the set of pixellocations represent a lattice or partition. For this reason, a samplingregion may also be called a sampling lattice.

In an alternative embodiment, the spatial locations of the pixelscontributing to at least two partitions change from one frame to thenext frame, and the sampling lattices are said to be three-dimensionalsampling lattices. Three-dimensional sampling lattices are performed bythe video partitioning module 16 by mapping a unique set of mtwo-dimensional sampling lattices to each successive set of tconsecutive frames of the input video signal.

The plural corresponding decimated frames output and identified by thevideo-partitioning module 16 exhibit a first temporal relationship, alsocalled a first relative temporal order, with respect to each other. Asdiscussed above, plural corresponding decimated frames are said to becorresponding if they originated from the same frame in the input videosignal. For the purposes of the present discussion, a temporalrelationship between a decimated frame in a first partition thatcorresponds to a decimated frame in a second partition may include anytime interval or set of time intervals separating the two correspondingdecimated frames. For example, data used to construct a first inputvideo frame, corresponding to a first segment, may be received by thevideo-partitioning module 16 during a first time interval, while dataused to construct a second video frame, corresponding to a secondsegment, is received during a second time interval. The first timeinterval and the second time interval may be separated by a third timeinterval. The resulting decimated frames obtained from the first inputvideo frame and the second input video frame and output by thevideo-partitioning module 18 may also be separated by the third timeinterval. A second relative temporal order (imparted by thetime-shifting module 18) is characterized by a selectively skewed thirdtime interval, as discussed more fully below.

An example first temporal relationship between frames of a video signalmay be such that video frames occur in a data stream in separate groupsof pixels that are grouped for transmission or reception in successivetime intervals. An example second temporal relationship between framesof a video signal may be such that video frames or portions thereof(e.g., latticed frames thereof) occur in a data stream in separategroups of pixels that are grouped for transmission or reception indifferent time intervals than those used for the first temporalrelationship.

In the above illustrative example, the first segment and the secondsegment correspond to individual video frames or video segments ofplural video frames. In practice, when an input video signal with inputframes is latticed, the first segment and the second segment includelatticed frames corresponding to sampling regions of one or more videoframes in the first segment and one or more video frames of the secondsegment of the input video frames, respectively. For the purposes of thepresent discussion, sampling regions of a video frame are also calledspatial portions.

Note that any number (p) of partitions of the video signal input to thevideo-partitioning module 16 may be identified by the video-partitioningmodule 16. The video-partitioning module 16 may include instructions forseparating an input video signal into plural data streams, where eachdata stream corresponds to one or more partitions, i.e., latticesderived from one or more corresponding sampling regions of the inputvideo signal.

The time-shifting module 18 implements instructions for selectivelychanging the first temporal relationship such that video data associatedwith a given frame is output from the time-shifting module 18 atdifferent time relative intervals than those characterizingcorresponding video data output from the video-partitioning module 16.For example, the amount of time used to transmit data corresponding to agiven frame in the video signal output by the video-partitioning module16 is different than the amount of time used to transmit data from thesame frame, after the time-shifting module 18 has processed the signaloutput by the video-partitioning module 16. This facilitates estimatinglost data (such as data lost due to burst errors during transmission) tofacilitate reconstructing video frames, as discussed more fully below.

For the purposes of the present discussion, a first segment of a videodata stream is said to be time shifted relative to a second segment ofthe video data stream when a predetermined time interval is insertedbetween transmission of the first segment and the second segment of thevideo data stream. For example, in certain embodiments discussed herein,corresponding segments of video sub-streams are time shifted relative toeach other to facilitate error concealment in a received video stream inthe event of a loss of video data for a predetermined data-lossinterval.

A video time-shifting module, such as the time-shifting module 18, maybe any hardware and/or software device, collection of devices, or otherentity that is adapted to move or rearrange in time different segmentsor other portions of a video. If the resulting time-shifted video weredisplayed before removal of the time-shifting, the display would bescrambled, as lattices of frames thereof would be received over a largertime span than ordinarily is allotted for receipt of a given frame. Inthe present specific embodiment, the time-shifting module 18 is adaptedto identify the plural streams and segments thereof of the video dataoutput from the video-partitioning module 16. The time-shifting module18 is further adapted to time shift one or more of the plural segmentsof the different streams with respect to one or more other of the pluralsegments of the different streams by one or more predetermined timeintervals.

In the present specific embodiment, the one or more predetermined timeintervals are sized to enable estimation of lost data packets of a videodata stream output by the encoder 20 for a predetermined data-lossinterval larger than approximately 500 milliseconds and less thanapproximately 2 seconds. A data-loss interval may be any time intervalduring which data in a data stream exhibits errors, is lost, corrupted,or is otherwise not available. Various mechanisms may cause data loss ina communications channel or network, including burst errors, signalfades, or other data-loss mechanisms.

The four parallel time-shifted data streams output by the time-shiftingmodule 18 are input to the encoder 20. Note that the paralleltime-shifted data streams (p video signals) may have been filtered viaan anti-aliasing filter implemented in the video-partitioning module 16.The anti-aliasing filter may be a lowpass filter or other type of filterdesigned to meet the anti-aliasing needs of a given implementation.

Parallel time-shifted data streams output by the time-shifting module 18are input to the video-compression module 22 of the encoder 20. Thevideo-compression module 22 may include instructions for compressing thefour time-shifted data streams. Exact details of compression algorithmsemployed by the video-compression module 22 are application specific andin accordance with a video coding specification, such as ISO/IEC MPEG-2Video (also known as ITU H.262) or ISO/IEC MPEG-4 Part 10 (also known asITU H.264).

Note that in a packet switched network, a single data stream lackingsub-streams will generally include plural packets of information, whereeach packet may take a different path through the network 32 to arriveat a destination address. Accordingly, for transmission over apacket-switched network, a single data stream lacking sub-streams may beany data stream where individual packets have similar source anddestination addresses. A data stream is said to have plural sub-streamstransmitted in parallel when data of the data stream is transmittedsimultaneously or approximately simultaneously from different sourceaddresses and/or transmitted to different destination addresses. Notethat the different source addresses and different destination addressesmay correspond to a single device when the single device has multipleaddresses associated therewith. For the purposes of the presentdiscussion, a packet-switched network may be any collection of one ormore communications links via which packets of information are routedvia address information associated with the packets.

The video compression module 22 outputs four parallel time-shifted andcompressed video data streams to the transmit chain 24. The transmitchain 24 includes various modules and functions used to prepare thecompressed video data output by the video compression module 22 fortransmission over the network 32. For example, the transmit chain 24includes the FEC module 26, which applies forward error correction toeach of the sub-streams output by the video-compression module 22.

FEC involves adding redundant data to a data stream to reduce oreliminate the need to retransmit data in the event of certain types ofdata loss. The redundant data facilitates reconstructing the data streamat the receiver in the event of data loss. The FEC module 26 addssufficient redundant data to each sub-stream output by thevideo-compression module 22 to enable the receivers 14, 30 to correctfor errors or data loss to each video sub-stream within anFEC-protection time interval, also called and FEC protect window.Generally, the FEC-protection time interval is often relatively smallcompared to a loss-concealment interval implemented by the LCALCs 44,54, as discussed more fully below.

Exact details of the transmit chain 24 are application specific. Forexample, when transmitting over a packet-switched network, such as theInternet, the transmit chain 24 may include a router and a firewallcoupled to an Internet Service Provider, and so on. When transmittingover a wireless network, the transmit chain 24 may include abaseband-to-IF (Intermediate Frequency) converter, automatic gaincontrol, filters, upconverters, a digital-to-analog converter, duplexer,antenna, and so on.

In the present specific embodiment, the transmit chain 24 transmits fourdata streams over the network 32. In other embodiments different numbersof streams may be used.

The network 32 may be implemented via a packet-switched network,circuit-switched network, wireless network, etc. Alternatively, thenetwork 32 may be replaced with a direct communications link between thetransmitter 12 and the receivers 14, 30. In wireless applications, thefour data streams may be transmitted to the receivers 14, 30 via fourdifferent communications channels or frequency bands.

The first receiver 14 receives or otherwise subscribes to all four datastreams transmitted by the transmit chain 24 via the network 32. Inwireless implementations, the first receive chain 36 may include one ormore amplifiers, frequency downconverters, filters, automatic gaincontrol circuits, IF-to-baseband converters, analog-to-digitalconverters, and so on. In packet-switched network applications, thereceive chain 36 or 46 may include one or more routers or other networkhardware to facilitate connecting the receiver 14 to the network 32.Exact details of the first receive chain 36 are application specific.Those skilled in the art with access to the present teachings mayreadily determine and implement a suitable receive chain to meet theneeds of a given application without undue experimentation.

In the present embodiment, the first receive chain 36 includes a reverseFEC module 38. The reverse FEC module 38 implements instructions forrepairing certain data loss or corruption occurring in one or more ofthe four data streams received from the transmit chain 24. The certaindata loss or corruption corresponds to data losses or corruption thatare within a predetermined data-loss interval, called the FEC protectwindow. Existing FEC modules, methods, and techniques may be readilyadapted for use with embodiments discussed herein by those skilled inthe art without undue experimentation. The first reverse FEC module 38is further adapted to undue any modifications to the parallel datastreams that were initially performed by the FEC module 26 of thetransmit chain 24 before the parallel data streams were transmitted overthe network 32.

The first video-decompression module 40 includes one or more circuits,routines, or instructions for decompressing the data streams that werecompressed by the video-compression module 22. The instructions mayinclude an inverse of the process used by the video-compression module22 to initially compress the data streams.

Decompressed video data streams are then output by thevideo-decompression module 40 and subsequently de-partitioned by thefirst video-de-partitioning module 42. The first video-de-partitioningmodule 42 includes instructions for removing any time shifting betweenvideo data streams that have been applied via the time-shifting module18 and for combining the data streams into a desired format inpreparation for input to LACL1 44.

Exact details of mechanisms for removing time-shifting and combiningdata streams are application specific. Various suitable methods may beemployed. For example, parallel video data streams may include tags orpacket headers that identify how the video data streams should bereconstructed. This identification information, which may be added bythe video partitioning module 16 and time-shifting module 18, may enablethe video-de-partitioning module 42 to recombine the data streams andremove time-shifting based on the tags or packet headers. The exactchoice and implementation of the identification information isapplication specific and may depend on the choice of data format. Forexample, MPEG-2 (Moving Picture Experts Group-2) formatted video mayemploy a program map or association table, which can includepacket-identification information to distinguish the p streams and toenable a receiver, such as the receiver 14 of FIG. 1, to undo the timeshifting and video partitioning performed by the time-shifting module 18and the video-partitioning module 16.

The first loss-concealment and latency-compensation module 44 includesinstructions for concealing any losses in the data that were notrepaired by the reverse FEC module 38. Furthermore, since differentpartitions or lattices (portions) of the original or input video signalinput to the video-partitioning module 16 were placed in separate datastreams and time shifted with respect to each other, a time interval,called a latency interval, may occur at the receivers 14, 30. Thislatency interval represents, for example, a time interval occurringbefore all portions of a set of initial video frames arrive at theloss-concealment and latency-compensation module 44. During this initiallatency interval, while the loss-concealment and latency-compensationmodule 44 waits for receipt of all portions of the initial data frames,the loss-concealment and latency-compensation module 44 conceals themissing information that has not yet been received.

The number of initial frames, for example, after a channel is changed ona television or set-top terminal, that will have missing data depends onthe amount of time-shift applied to each data stream by thetime-shifting module 18 and the sizes of the segments (e.g., 2-secondsegments) in which the input video signal has been divided by thevideo-partitioning module 16. For example, if a three-second intervaloccurs between transmission of a first segment of a first data streamand transmission of a corresponding first segment of a fourth datastream, then approximately three-seconds worth of initial frames will bereceived by the LACL1 44 before latency-compensation is no longer neededfor the video. The output of the LACL1 44 may be input to another stageof video processing, to a display device, to memory, or to anotherentity.

Various methods for concealing missing or lost information may beemployed by the loss-concealment and latency-compensation module 44. Forexample, in one implementation, missing pixel information is estimatedvia an interpolation process. The interpolation process may includeperforming linear or nonlinear interpolation in a direction across avideo frame that exhibits the least amount of change in colorbrightness, and/or combination thereof. Providing missing or corruptedpixel information is a type of upsampling. Accordingly, the upsamplingof missing or corrupted pixels may be performed by filling in pixels inthe direction of decreasing luma and/or chroma gradients using nonlinearupsampling.

Interpolation may include determining how certain information variesacross a video display and then continuing or interpolating the patternto fill in missing pixels. Various types of interpolation are possible.Details for determining values for missing pixel information can beapplication specific. Those skilled in the art with access to thepresent teachings may readily implement different embodiments of theinvention. For example, any suitable concealment algorithm can be usedto estimate or derive missing pixels.

Furthermore, while the LACL1 44 generally employs pixel informationassociated with a given frame to estimate lost pixel information withinthe frame, embodiments are not limited thereto. For example, in certainimplementations, pixel information from adjacent frames may be employedto further estimate lost pixel information in a given frame.Furthermore, loss-concealment techniques may further include employingmotion information, such as via motion vectors, to estimate lost pixelinformation.

The second receiver 30 operates similarly to the first receiver 14 withthe exception that the second receiver 30 subscribes to only a subset ofthe parallel data streams transmitted by the transmit chain 24. In thiscase, the second loss-concealment and latency-compensation module 54conceals data corresponding to the missing data stream. In the presentembodiment, this would correspond to one pixel for each predeterminedgroup of four (in the specific case where p=4) adjacent pixels in eachvideo data frame.

Note that embodiments discussed herein are not limited to a particularvideo format. Different video formats may employ different encoders anddecoders. Furthermore, embodiments are not limited to video datatransmission, as similar concepts discussed herein may be employed forrobust transport of audio data or other types of data. Those skilled inthe art with access to the present teachings may readily modify themodules of the system 10 to meet the needs of a given implementationwithout undue experimentation.

In the present specific embodiment, each pixel in a video frame includesa luma component (Y′), a chroma-blue component (Cb), and a chroma-redcomponent (Cr). The Y′ component contains brightness information, whilethe chroma components Cb and Cr contain color information.

Certain video formats, such as MPEG-2 use chroma subsampling. Chromasubsampling involves allocating more resolution and associated bandwidthto luma components than chroma components, since human visual perceptionis less sensitive to chroma information than it is to brightnessinformation, i.e., luma or luminance information.

In certain applications using video formats employing chromasubsampling, chroma information for a given video frame may be sentseparately from luma information. Hence, a given frame may be sent asseparate sub-frames, where the chroma sub-frames are smaller than theluma sub-frames by a factor determined by a subsampling ratio employedwhen formatting the video.

Hence, a given video frame may be transmitted in separate sub-frames,where each separate sub-frame contains a component of the combinedframe. In such applications, the separate sub-frames may be shuffled intime and transmitted in accordance with an embodiment discussed herein.If, for example, portions of a luma frame are lost due to a large signalfade or interference larger than the FEC protect window, and thecorresponding chroma sub-frames are not lost, the LCALCs 44, 54 mayreadily estimate the lost pixels of the luma sub-frames. The resultingestimated picture may represent a relatively accurate visual depictionof the original or input video frame, as the existing chroma informationmay facilitate disguising the lost luma information. Hence, ifcomponents of any sub-frames are lost, the remaining sub-frames mayfacilitate disguising or concealing missing pixel information. Note thatvideo formats not employing chroma subsampling and not employingsub-frames may be employed without departing from the scope of thepresent teachings.

While the embodiment of FIG. 1 has been primarily discussed with respectto processing of a video signal comprising a sequence of video frames,embodiments are not limited thereto. For example, an audio signal orother media signal having an output order of its samples or elements maybe separated into different partitions in accordance with the presentteachings, where the different partitions of the audio or media signalare processed and shuffled in time and optionally transmitted viadifferent parallel data streams before they are recombined at areceiver.

FIG. 2 is a diagram illustrating a first example partitioning of a videoframe 60 by the system 10 of FIG. 1. The frame 60 includes rows andcolumns of pixels, where the location of the pixel in the i-th row andj-th column is denoted ij. Each pixel ij in the frame 60 is associatedwith a given sampling region, also called sampling lattice, which isassociated with a partition, to be processed and transmitted as aseparate corresponding video stream. Each video stream corresponds to adistinct lattice or partition. In one embodiment, the plural separatevideo streams are multiplexed and transported over a single transportstream over a single transmission channel. In an alternative embodiment,the plural video streams are multiplexed in a single video stream withauxiliary information that identifies the respective partitions withinthe video stream. The respective partitions are shuffled within thevideo stream according to a first relative temporal order such that thecorresponding frames in different partitions are transmitted atstrategically different times to facilitate error concealment.

In the present embodiment, the video frame 60 is logically partitionedby mapping non-overlapping contiguous 2×2 matrices on the frame 60,where each pixel from each 2×2 matrix is assigned to a respectivepartition, i.e., lattice. For example a first group of pixels in thetop-left matrix includes pixels 00, 01, 10, and 11. Pixel 00 is assignedto a first partition, V0. Pixel 01 is assigned to a second partition,V1. Pixel 10 is assigned to a third partition, V2, and pixel 11 isassigned to a fourth partition, V3. Note that the different partitionsV0-V3 represent data sent via respective streams S0-S3.

Hence, the partition V0 is assigned every other pixel on every other rowstarting with pixel 00, i.e., V0 is assigned pixels mn, where m and nare even integers. V1 is assigned pixels mw, where m is an even integer,and w is an odd integer. V2 is assigned pixels qn, where q is an oddinteger, and n is an even integer. Similarly, V3 is assigned pixels qw,where q and w are odd integers. Consequently, each 2×2 matrix containsfour pixels that are each assigned to one of four different lattices.Note that in FIG. 2, pixels labeled V0 represent a zeroth latticed frame(LV0); pixels labeled V1 represent a first latticed frame (LV1); pixelslabeled V2 represent a second latticed frame (LV2), and pixels labeledV3 represent a third latticed frame (LV3). When referring to acollection of frames in a video signal, pixels labeled V0-V1 representpartitions or lattices of a video signal comprising the collection offrames, and such lattices are sent via corresponding video data streamsS0-S3, as discussed more fully below.

FIG. 3 a is a first example timing diagram 70 (with p=4) illustratingtransmission of time-shifted video streams S0-S3, which correspondrespectively to the compressed versions of video partitions V0-V3 by theexample communications system 10 of FIG. 1. Interval T_(L) is an exampleof a data loss occurrence. The timing diagram 70 includes a horizontaltime axis 72. In an alternate embodiment, T_(L) or any interval on thehorizontal axis 72 may represent an interval of multiplexed portions ofS0-S3, such as when respectively corresponding portions of S0-S3 aremultiplexed over an interval in a transport stream. In one embodiment,corresponding portions of S0-S3 may be multiplexed over an interval in avideo stream. An interval may be perceived in FIG. 3 a as a verticalslice through S0-S3. For a stream carrying multiplexed portions ofS0-S3, a particular interval may be at a corresponding location of thestream and have a corresponding width. A particular interval may have acorresponding starting time and a corresponding duration. In oneembodiment, a particular interval contains at most one provided portionfrom each of the streams S0-S3 and each respective provided portion inthe particular interval is from at most one compressed picture.

In one embodiment, a portion of a compressed picture of a video stream(e.g., S0) may be provided over an interval while providing one or moreportions from the other remaining streams (e.g., S1-S3), where each ofthe one or more provided portions corresponds to: (1) at most onecompressed picture, and (2) at most one stream from S1-S3. Foursimultaneously provided compressed pictures over a first interval, ortheir respective corresponding provided portions, may be respectivelyassociated with a different temporal or display time of the input videosignal (i.e., the four pictures are non-corresponding or non-associatedpictures since they did not originate from the same picture of the inputvideo signal). Two or more provided compressed pictures over a secondinterval, or their respective corresponding provided portions, may becorresponding or associated pictures (i.e., they are corresponding orassociated pictures since they originated from the same picture of theinput video signal).

Various video segments (t0-t5) are labeled in FIG. 3 a based on thetemporal progression of each video stream S0-S3. Video segments arelabeled in accordance with the temporal progression of successivesegments of pictures of the input video signal to the transmitter 12 ofFIG. 1. A given segment, such as t1, represents corresponding compressedvideo frames, i.e., processed decimated video frames that originatedfrom the same segment of frames of the input video signal to the system10 of FIG. 1. As described previously, a video segment comprises nfconsecutive compressed video frames in each of the video streams S0-S3.Note that the corresponding compressed segments, or correspondingpictures in each segment, have a common display or output time. In FIG.3 a, a set of corresponding video segments is depicted by all of theinstances of tk, where: k is an integer. The labels tk demarcatesegments corresponding to time intervals in the temporal progression ofeach video stream, and each instance of tk represents a correspondingset of nf compressed video frames of a video segment, where nf can havea value equal or greater than one. For example, the time intervalslabeled t1 in the temporal progressions of video streams S0-S3 pertainto corresponding video segments, each with nf compressed video frames.

In the present embodiment, each video stream S0-S3 is shifted in timerelative to the other video streams S0-S3 as shown in FIG. 3 a. In thisexample embodiment, the segments t0-t5 are approximately two secondslong, and successive segments are offset or skewed with respect to eachother, for example, by approximately 200 milliseconds, which correspondsto the FEC protect window employed by the FEC module 26 of FIG.

With reference FIGS. 1 and 2, if the frame 60 shown in FIG. 2 occurswithin a certain time interval t3 of its corresponding picture in theinput video signal to transmitter 12 of FIG. 1, the frame 60 is said tooccur within the set of corresponding video segments associated with thet3 time interval, or herein the t3 segment of video or t3 segment.However, with reference to FIG. 3 a, V0-V3 lattices corresponding to theset of t3 corresponding segments are transmitted time-shifted or delayedas shown by the staggering of t3 in FIG. 3 a. In one embodiment, theexact delay between successive t3 segments is slightly less than t3, andis approximately t3 minus the 200 millisecond FEC protect intervalwindow. This delay may be according to the number of frames, nf, insegment t3, such that the delay represents an integral number of frametransmission intervals, where a frame transmission interval representsthe time used to transmit a single frame.

Over a transmission time interval that can be depicted as a verticalslice in FIG. 3 a that spans four segments (e.g., t0-t3), four (p=4)non-corresponding segments (where non-corresponding segments areassigned different labels, e.g., t0-t3), one being a t3 segment, may be,in an alternative implementation, multiplexed onto a single transportstream and then transmitted by transmitter 12 as opposed to beingtransmitted via multiple streams. Transmitter 12 may provide over aparticular interval a portion of a single picture from stream S0 whileproviding a corresponding portion of a single picture from one of theremaining streams, S1-S3. Transmitter 12 may provide over an intervalmultiplexed portions of S0-S3 such that at most one portion from each ofthe streams S0-S3 is provided, and such that each provided portion inthe interval is from at most one compressed picture.

The receiver 14 of FIG. 1 will wait to receive the t3 segments from thedata streams S0-S3 before a given frame is considered complete. Beforeall t3 segments for each data stream S0-S3 are received for a givenframe, the LCALC1 44 of FIG. 1 may interpolate to estimate pixelinformation that has not yet arrived.

After an initial latency period, subsequent frames will have alreadyarrived for the next frame so that display or output of subsequentframes will not require latency compensation. This is called apipelining effect, where successive video segments are sent back toback. Note that the various video segments t0-t3 are sent pipelined andin parallel. Note that overlap occurs during successive transmission ofthe segments t0-t3 for different data streams S0-S3.

The video data streams S0-S3 are said to be time-shifted with respect toeach other, since the different video segments t0-t3 for different datastreams S0-S3 are transmitted at different times, i.e., at shiftedtimes, rather than at similar times.

FIG. 3 b is a second example timing diagram 71 illustrating exampletransmission timing of video data from a group of video frames (P_(k) .. . P_(k+w)) 73 corresponding to the segments t2 of FIG. 3 a. In thepresent example, the group of video frames 73 includes w frames,including a kth frame (P_(k)) and a (k+w)th frame (P_(k+w)). Note thatin the present example, w frames is equivalent to 2 seconds worth offrames. However, note that differently sized segments may be used. Forexample, in an alternative implementation where w=1, each of thesegments t0 . . . t5 of FIG. 3 a represent a time interval needed totransmit one frame worth of data. Generally, in the embodimentsdiscussed herein, the temporal widths of the segments t0 . . . t5 ofFIG. 3 a are chosen to correspond to an integral number of frames.

The initial input group of video frames 73 are partitioned intolattices, including a zeroth group of latticed frames (LV0(k) . . .LV0(k+w)) 75, a first group of latticed frames (LV1(k) . . . LV1(k+w))77, a second group of latticed frames (LV2(k) . . . LV2(k+w)) 79, and athird group of latticed frames (LV3(k) . . . LV3(k+w)) 81. The latticedframes LV0 . . . LV3 are also called decimated or down-sampled framesand include pixels labeled V0-V3 . . . , respectively. The latticedframes LV0 . . . LV3 are transmitted via data streams S0-S3,respectively, at strategically staggered time intervals. Alternatively,the data streams S0-S3 are multiplexed onto a single transport stream,which may include additional information, such as audio synchronizationinformation, closed caption information, and so on. Close-captioninformation may be included for each respective decimated or sub-streamversion of the video stream, albeit redundant. Exact details forimplementing transport streams for a given application may be readilydetermined by those skilled in the art. For illustrative purposes, thesegments in FIG. 3 b are shown corresponding to the t2 segments of FIG.3 a.

Hence, one example embodiment involves transmitting different latticesof one or more video frames via different data streams via selectivelystaggered time intervals (segments), where each latticed frame is chosenby selecting one pixel from each matrix of the frame. In other words,video frames are partitioned into latticed frames; each latticed frameis assigned to a different data stream; then transmission of thedifferent latticed frames is selectively staggered in time. Thestaggering may be chosen based on characteristics of a giventransmission medium, such as the types and durations of burst errors towhich the transmission medium is prone.

FIG. 4 is diagram of the example video frame of FIG. 2 showing a firstexample data-loss pattern 80 for the data-loss interval T_(L) of FIG. 3a corresponding to the first video segment t1.

With reference to FIGS. 3 and 4, for illustrative purposes, the frame 60is assumed to have lost, during the data-loss interval T_(L) of FIG. 3,portions of the zeroth data stream S0 and the first data stream S1occurring in the video segment t1. Note that for the video segment t1,only data from data streams S0 and S1 are lost, i.e., occur within thedata-loss interval T_(L). Consequently, the frame 60 will only lose aportion of the pixels during the relatively large data-loss intervalT_(L), which may be larger than 500 milliseconds, and is approximatelytwo seconds in the present example. Component values for the lost pixelsmay be interpolated or otherwise estimated, thereby avoiding ablack-screen effect that might otherwise occur for such a largedata-loss interval.

As shown in FIG. 4 pixel information for pixels labeled V0 and V1 andcorresponding to the data streams S0 and S1, respectively, are lost forthe video segment t1 during the large data-loss interval T_(L). The losspattern 80 includes horizontal bands 82 of lost pixels.

FIG. 5 is diagram of a second example video frame 90 partitioned inaccordance with the video partitioning illustrated in FIG. 2 and showinga second example data-loss pattern 100 for the data-loss interval T_(L)of FIG. 3 a for the segment t2.

With reference to FIGS. 3 and 5, the second example video frame 90represents a video frame associated with the second video segment t2,which illustrates portions of data streams S1 and S2 that are lostduring the data-loss interval T_(L) shown in FIG. 3. Note that in FIG.3, the video segment t2 occurs in the streams S1 and S2 during thedata-loss interval T_(L), and corresponding portions (pixels labeled V1and V2) of the streams S1 and S2 are considered lost for the purposes ofthe present example.

Accordingly, pixels associated with data streams S1 and S2 are shownmissing in FIG. 5, forming diagonal bands 102 of lost pixels.

FIG. 6 is diagram of a third example video frame 110 partitioned inaccordance with the video partitioning illustrated in FIG. 2 and showinga third example data loss pattern 120 for the data-loss interval T_(L)of FIG. 3 a for a third video segment t3.

With reference to FIGS. 3 and 6, data loss corresponding to thedata-loss interval T_(L) affects the third the third video segment t3 ofthe second data stream S2 and the third data stream S3. Accordingly, thethird data-loss pattern 120 includes horizontal bands 112 of missing orcorrupted pixels corresponding to the pixels labeled V2 and V3, whichcorrespond to the data streams S2 and S3.

Note that the receivers 14, 30 of FIG. 1 are adapted to selectivelycombine the video data streams S0-S3 and to synchronize or temporallyalign the streams so that the various segments t0-t5 of each data streamapproximately temporally coincide in preparation for display or output.For example, the t2 segment components of the data streams S0-S3 mayapproximately coincide with each other after processing by the receivers14, 30, thereby facilitating reconstructing and displaying the videoframe or frames associated with the segment or time interval identifiedby t2.

Note that the video-data loss corresponding to the data-loss intervalT_(L) of FIG. 3 a is spread over five intervals or video segments t0-t4.The five intervals may represent five frames in implementations whereone frame is allocated for each interval t0-t4. The five intervals orframes include the frames 60, 90, 110 of FIGS. 4-6 for the videosegments t1-t3.

While six video segments t0-t5 are shown in FIG. 3, embodiments are notlimited thereto. For example, the data streams S0-S3 may be partitionedinto more or fewer than six segments. Furthermore, the segments t0-t5may be shuffled so that the segments occur in a different order ortiming than shown in FIG. 3.

Note that losses that occur in successive time intervals or videosegments t0-t3 occur in different patterns, thereby facilitatingdisguising losses. For example, after predetermined time interval, thedata loss pattern changes. If the loss patterns shown in FIGS. 4-6 aresuccessively displayed, the resulting data loss may be more difficult toperceive than if a single data-loss pattern remained during all timeintervals.

FIG. 7 is a diagram illustrating a second example partitioning of afourth video frame 130 by the system 10 of FIG. 1. The fourth videoframe 130 is logically partitioned into groups, i.e., matrices, of fourlinearly disposed adjacent pixels, where each pixel, labeled V0-V3, froma 1×4 matrix of pixels is assigned a different video-data stream S0-S3,respectively. The fourth video frame 130 is considered logicallypartitioned into matrices of four pixels, since in practice, the system10 need not actually separate or partition a frame into the matrices.Instead, the system 10 of FIG. 1 may merely assign certain pixels todifferent data streams based on a predetermined methodology withoutphysically or electronically grouping or separating the pixels in 1×4matrices as shown in FIG. 7.

For example, pixels 00, 01, 02, and 03 form a first matrix of pixels,where pixel 00 is assigned to data stream S0; 01 to data stream S1; 02to data stream S2; 03 to data stream S3. The pixel-grouping pattern,i.e., matrix pattern, repeats as shown in FIG. 7. In general, pixels fordata streams S0 and S2 are selected from alternating pixels in everyother column of the video frame 130. Pixels for data streams S1 and S3are also selected from alternating pixels in every other column. Thisgrouping of pixels ensures that for a given loss interval, such as thedata-loss interval T_(L) of FIG. 3, that for any frame that is missingtwo data streams, the resulting lost pixels will not occupy an entireline or column. This may ensure that the data losses will be lessperceptible to the human eye than they would be if entire lines orcolumns of pixels were lost. This is based on the notion that humanperception is more sensitive data loss occurring along to horizontal orvertical lines than along diagonal lines.

With reference to FIGS. 2, 3 a, and 7 the logical partitioning of theframe 130 of FIG. 7 into 1×4 adjacent matrices four pixels results indifferent pixel assignments to different data streams than results fromthe partitioning shown in FIG. 2. Consequently, the corresponding dataloss patterns for the example data-loss interval T_(L) of FIG. 3 a willbe different than the corresponding data-loss patterns 80, 100, 120 ofFIGS. 4-6.

FIG. 8 is diagram of the fourth example video frame 130 of FIG. 7showing a fourth example data-loss pattern 140 for the data-lossinterval T_(L) of FIG. 3 a for the first video segment t1.

With reference to FIGS. 3 and 8, data in FIG. 8 is lost from datastreams S0 and S1 for the first video segment t1. Accordingly, forillustrative purposes, pixels (labeled V0 and V1) corresponding to datastreams S0 and S1 are marked in FIG. 8 as having been lost. Theresulting loss pattern 140 includes alternating groups of two pixels142, which form so-called horizontally biased paired diagonal loss.

FIG. 9 is diagram of a fifth example video frame 150 partitioned inaccordance with the video partitioning illustrated in FIG. 7 and showinga fifth example data-loss pattern for the data-loss interval T_(L) ofFIG. 3 a for the second video segment t2.

With reference to FIGS. 3 a and 9, data is lost from data streams S1 andS2 for the second video segment t2 during the data-loss interval T_(L)of FIG. 3 a. This yields the fifth data-loss pattern 160, which includesmissing pixels for the data streams S1 and S2. The resulting losspattern 160 includes alternating pairs of missing or corrupted pixels162 (labeled V1 and V2), which form horizontally biased paired diagonalloss.

FIG. 10 is diagram of a sixth example video frame 170 partitioned inaccordance with the video partitioning illustrated in FIG. 7 and showinga sixth example data-loss pattern 180 for the data-loss interval T_(L)of FIG. 3 a for the third video segment t3.

With reference to FIGS. 3 a and 10, data is lost from data streams S2and S3 for the third video segment t3 during the data-loss intervalT_(L) of FIG. 3 a. This results in the sixth data-loss pattern 180,which includes the horizontally biased paired diagonal loss of pixels182 (labeled V2 and V3).

FIG. 11 is a diagram illustrating a third example partitioning of aseventh video frame 190 by the system 10 of FIG. 1. The seventh videoframe 190 is logically partitioned into 2×2-pixel groups of four pixels.Groupings, i.e., matrices of pixels labeled V0-V3, as shown in FIG. 11and 12 are similar to those shown in FIG. 2 with the exception thatmatrices on adjacent pairs of rows are offset by one pixel.

FIG. 12 is a third example timing diagram illustrating paralleltransmission of time-shifted video data streams S0, S1′, S2′, and S3 andan example data-loss interval T_(L), where each stream S0, S1′, S2′, andS3 corresponds to a partition or lattice of the video in the examplecommunications system 10 of FIG. 1.

The third example timing diagram 200 is similar to the first exampletiming diagram 70 of FIG. 3 a with the exception that the first videodata stream S1′ of FIG. 12 is segmented (via positioning of t0-t5)similarly to S2 of FIG. 3, and S2′ of FIG. 11 is segmented similarly toS1 of FIG. 3.

Note that the way a given video data stream is segmented may be definedrelative to the other video data streams. For example, video segmentst0-t5 for the different data streams S0, S1′, S2′, and S3 may betransmitted with predetermined temporal relationships. In FIG. 12, thistemporal relationship is partially described by a time offset ofapproximately 1800 milliseconds between occurrences of the correspondingvideo segment component in an adjacent video data stream. For example,the video segment component t2 of the data stream V2′ beginsapproximately 1800 milliseconds after the start of the video segmentcomponent t2 of the adjacent data stream S1′.

FIG. 13 is diagram of the seventh example video frame 190 partitioned inaccordance with the video partitioning illustrated in FIGS. 11 and 12and showing an eighth example data-loss pattern 210 for the data-lossinterval T_(L) of FIG. 12 for the first video segment t1.

With reference to FIGS. 12 and 13, data in FIG. 12 is lost from datastreams S0 and S2′ for the first video segment t1. This results in theseventh example data-loss pattern 210. The seventh example data-losspattern 210 includes alternating pairs of vertical pixels 212, whereinalternating pairs of rows have alternating lost 2×1 pairs of pixels thatare offset by one pixel relative to adjacent pairs of rows. The seventhexample data-loss pattern 210 is said to exhibit vertically biasedpaired diagonal loss, since the vertical 2×1 pixel pairs occurdiagonally across the seventh example video frame 190.

FIG. 14 is diagram of an eighth example video frame 220 partitioned inaccordance with the video partitioning illustrated in FIGS. 11 and 12and showing an eighth example data-loss pattern 230 for the data-lossinterval T_(L) of FIG. 12 for the second video segment t2.

With reference to FIGS. 12 and 14, data streams S1′ and S2′, whichinclude pixels labeled V1′ and V2′, respectively, experience data lossduring the data-loss interval T_(L) of FIG. 12 for the second videosegment t2. Hence, the eighth data-loss pattern 230 shows missing orcorrupted data 232 corresponding to data streams S1′ and S2′. Themissing or corrupted data 232 exhibits vertically biased paired diagonalloss.

FIG. 15 is diagram of a ninth example video frame 240 partitioned inaccordance with the video partitioning illustrated in FIGS. 11 and 12and showing a ninth example data-loss pattern 250 for the data-lossinterval T_(L) of FIG. 12 for third video segment t3.

With reference to FIGS. 12 and 15, the data streams S1′ and S3, whichinclude pixels labeled V1′ and V3, respectively, experience data lossduring the data-loss interval T_(L) of FIG. 12 for the third videosegment t3. This results in the ninth data-loss pattern 250, whichincludes missing or corrupted vertical pairs 252 of pixels exhibitingvertically biased paired diagonal loss.

FIG. 16 is a flow diagram of a first example method 260 suitable for usewith the communications system 10 of FIG. 1. The method 260 includes afirst step 262, which includes obtaining data.

A second step 254 includes partitioning the data into identifiableportions of data, also called partitions or lattices. With reference toFIG. 3 a, example identifiable portions of data may include the datastreams S0-S3 and the accompanying segments t0-t5. Note thatidentifiable portions of data may include more or fewer than 4 datastreams without departing from the scope of the present teachings.

A third step 266 includes selectively shifting one or more of theidentifiable portions of data relative to one or more other identifiableportions of the data, resulting in shifted partitioned data in responsethereto. The data streams S0-S3 of FIG. 3 a represent shifted portionsof data, where the segments t0-t5 are transmitted at different times inthe different data streams S0-S3.

For the purposes of the present discussion, shifting of a first portionof data relative to another potion of data may mean adjusting the firstor second portions of data so that when the portions of data aretransmitted or sent, they are transmitted or sent at a different timesrelative to each other than they would be if they were not shifted. Forexample, if a first portion and a second portion of data are typicallytransmitted approximately simultaneously, such that correspondingsegments (e.g. segments identified by t0-t5 in FIG. 3 a) of the portionsare temporally aligned, shifting of the first portion of data relativeto the second portion of data would result in transmission of thecorresponding segments at different times, such that the segments arenot temporally aligned.

A fourth step 268 includes transmitting the shifted and partitioneddata. The data may be transmitted over a network, wirelesscommunications channel, or via another medium.

FIG. 17 is a flow diagram of a second example method 260 suitable foruse with the communications system 10 of FIG. 1. The second examplemethod 260 includes a partitioning step 272, which includes partitioningvideo data into plural partitions that correspond to different latticesor sampling regions of a displayed video.

A subsequent shifting step 274 includes selectively shifting one or moreof the partitions in time relative to one or more other partitions andproviding one or more data streams that include(s) time-shiftedpartitions in response thereto.

Next, an FEC step 226 includes applying forward error correction to oneor more of the time-shifted partitions. This results in one or moretime-shifted partitions of video data and redundant data incorporatedfor implementing forward error correction to protect the video data whentransmitted.

A subsequent transmission step 278 includes transmitting the one or moretime-shifted partitions of video data to which forward error correctionhas been applied over a packet-switched network or other medium.

Note that various steps of the methods 260 and 270 may be replaced withother steps. In addition, certain steps may be interchanged with othersteps, and additional or fewer steps may be included. For example, themethod 270 could include applying anti-aliasing filtration to the videodata, compressing the video data, and so on. In addition, a receivingstep may be included, wherein transmitted video data is received, andany data losses in the received video data that are not repaired by FECprocessing may be concealed via interpolation techniques. In addition, areceiver may subscribe to all of the identifiable partitions of videodata or a subset thereof. Furthermore, chroma pixel information may beseparated from luma pixel information in a given frame and treatedseparately. Other modifications to the methods 260 and 270 of FIGS. 15and 16 may be made without departing from the scope of the presentteachings.

An alternative method includes partitioning an initial data streamcorresponding to an input video signal into two or more data streams orother identifiable partitions or portions as discussed herein. The datastreams may then each be selectively assigned different Scalable VideoCoding (SVC) enhancement layers. Alternatively, a first set of datastreams is implemented as discussed herein, and a second set of datastreams is implemented as a set of SVC enhancement layers.

While certain embodiments discussed herein are discussed primarily withrespect to the processing and transport of video data, embodiments arenot limited thereto. For example, other types of data, such as audiodata, text, or other types of data may be partitioned, shifted, andtransmitted in accordance with the present teachings without departingfrom the scope thereof.

While various embodiments disclosed herein have has been discussed withrespect to creation of four data streams from a single initial datastream corresponding to an input video signal, embodiments are notlimited thereto. For example, certain embodiments may partition aninitial data stream corresponding to an input video signal into two ormore data streams that are transmitted in parallel. Alternatively,certain portions of an initial data stream corresponding to an inputvideo signal may be selectively rearranged and transmitted in a singledata stream rather than in separate parallel data streams withoutdeparting from the scope of the present teachings. Furthermore theinitial data stream corresponding to an input video signal itself mayinclude plural parallel data streams rather than a single data streamlacking sub-streams.

Although a process of the present invention may be presented as a singleentity, such as software, instructions, or routines executing on asingle machine, such software, instructions, or routines can readily beexecuted on multiple machines. That is, there may be multiple instancesof a given software program, a single program may be executing on two ormore processors in a distributed processing environment, parts of asingle program may be executing on different physical machines, etc.Furthermore, two different programs, such as an FEC and an LCALCprogram, can be executing in a single machine, or in different machines.A single program can be operating as an FEC for one data-processingoperation and as a LCALC for a different data-processing operation.

Although the invention has been discussed with respect to specificembodiments thereof, these embodiments are merely illustrative, and notrestrictive, of the invention. Embodiments of the present invention canoperate between any two processes or entities including users, devices,functional systems, or combinations of hardware and software. Forexample, while partitioning has been described herein as operatingprimarily upon video frames, other portions, arrangements or groupingsof video can be subjected to partitioning. For example, groups ofpictures (GOPs), pictures, frames, or other layers or portions of videocontent may be subjected to partitioning.

This application is related to U.S. patent application Ser. No. ______,entitled “TIME-SHIFTED TRANSPORT OF MULTI-LATTICED VIDEO FOR RESILIENCYFROM BURST-ERROR EFFECTS”, filed concurrently and U.S. patentapplication Ser. No. ______, entitled “PROCESSING OF IMPAIRED ANDINCOMPLETE MULTI-LATTICED VIDEO STREAMS”, filed concurrently, both ofwhich are incorporated by reference in their entirety for all purposes.

Any suitable programming language can be used to implement the routinesor other instructions employed by various network entities. Exampleprogramming languages include C, C++, Java, assembly language, etc.Different programming techniques can be employed such as procedural orobject oriented. The routines can execute on a single processing deviceor multiple processors. The routines can operate in an operating systemenvironment or as stand-alone routines occupying all, or a substantialpart, of the system processing.

In the description herein, numerous specific details are provided, suchas examples of components and/or methods, to provide a thoroughunderstanding of embodiments of the present invention. One skilled inthe relevant art will recognize, however, that an embodiment of theinvention can be practiced without one or more of the specific details,or with other apparatus, systems, assemblies, methods, components,materials, parts, and/or the like. In other instances, well-knownstructures, materials, or operations are not specifically shown ordescribed in detail to avoid obscuring aspects of embodiments of thepresent invention.

A “processor” or “process” includes any human, hardware and/or softwaresystem, mechanism or component that processes data, signals or otherinformation. A processor can include a system with a general-purposecentral processing unit, multiple processing units, dedicated circuitryfor achieving functionality, or other systems. Processing need not belimited to a geographic location, or have temporal limitations. Forexample, a processor can perform its functions in “real time,”“offline,” in a “batch mode,” etc. Portions of processing can beperformed at different times and at different locations, by different(or the same) processing systems. A computer may be any processor incommunication with a memory.

Reference throughout this specification to “one embodiment”, “anembodiment”, “a specific embodiment”, of “an implementation” means thata particular feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodiment ofthe present invention and not necessarily in all embodiments. Thus,respective appearances of the phrases “in one embodiment”, “in anembodiment”, or “in a specific embodiment” in various places throughoutthis specification are not necessarily referring to the same embodiment.Furthermore, the particular features, structures, or characteristics ofany specific embodiment of the present invention may be combined in anysuitable manner with one or more other embodiments. It is to beunderstood that other variations and modifications of the embodiments ofthe present invention described and illustrated herein are possible inlight of the teachings herein and are to be considered as part of thespirit and scope of the present invention.

Embodiments of the invention may be implemented in whole or in part byusing a programmed general purpose digital computer; by usingapplication specific integrated circuits, programmable logic devices,field programmable gate arrays, optical, chemical, biological, quantumor nanoengineered systems or mechanisms; and so on. In general, thefunctions of the present invention can be achieved by any means as isknown in the art. Distributed or networked systems, components, and/orcircuits can be used. Communication, or transfer of data may be wired,wireless, or by any other means.

It will also be appreciated that one or more of the elements depicted inthe drawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application. It isalso within the spirit and scope of the present invention to implement aprogram or code that can be stored in a computer-readable storage mediumor device to permit a computing system to perform any of the methodsdescribed above.

Additionally, any signal arrows in the drawings/figures should beconsidered only as examples, and not limiting, unless otherwisespecifically noted. For example, an arrow on a signal path indicatingcommunication in one direction does not necessitate that communicationalong that signal path is limited to that one direction.

Furthermore, the term “or” as used herein is generally intended to mean“and/or” unless otherwise indicated. In addition, the term “includes” istaken to mean “includes but is not limited to.” Combinations ofcomponents or steps will also be considered as being noted, whereterminology is foreseen as rendering the ability to separate or combineis unclear.

As used in the description herein and throughout the claims that follow“a”, “an”, and “the” include plural references unless the contextclearly dictates otherwise. Furthermore, as used in the descriptionherein and throughout the claims that follow, the meaning of “in”includes “in” and “on” unless the context clearly dictates otherwise.

The foregoing description of illustrated embodiments of the presentinvention, including what is described in the Abstract, is not intendedto be exhaustive or to limit the invention to the precise formsdisclosed herein. While specific embodiments of, and examples for, theinvention are described herein for illustrative purposes only, variousequivalent modifications are possible within the spirit and scope of thepresent invention, as those skilled in the relevant art will recognizeand appreciate. As indicated, these modifications may be made to thepresent invention in light of the foregoing description of illustratedembodiments of the present invention and are to be included within thespirit and scope of the present invention.

Thus, while the present invention has been described herein withreference to particular embodiments thereof, a latitude of modification,various changes and substitutions are intended in the foregoingdisclosures, and it will be appreciated that in some instances somefeatures of embodiments of the invention will be employed without acorresponding use of other features without departing from the scope andspirit of the invention as set forth. Therefore, many modifications maybe made to adapt a particular situation or material to the essentialscope and spirit of the present invention. It is intended that theinvention not be limited to the particular terms used in followingclaims and/or to the particular embodiment disclosed as the best modecontemplated for carrying out this invention, but that the inventionwill include any and all embodiments and equivalents falling within thescope of the appended claims.

1. A method for receiving and outputting plural representations of avideo signal in a video stream, the method comprising: receiving a videostream, wherein the video stream includes plural representations of avideo signal, wherein each of the plural representations of the videosignal (PROTVS) includes a respective sequence of processed pictures,wherein each processed picture of each of the PROTVS represents acorresponding respective picture of the video signal, wherein eachrespective picture of the video signal is represented by at most oneprocessed picture of each of the PROTVS, and wherein a first pluralityof the processed pictures of a second representation of the video signal(SROTVS) is received in the video stream after at least a portion of afirst plurality of the processed pictures of a first representation ofthe video signal (FROTVS); determining plural sets of one or moreassociated processed pictures of the PROTVS, wherein each processedpicture in each determined set of one or more associated processedpictures (DSOOOMAPP) represents the same respective picture of the videosignal, wherein each respective DSOOOMAPP includes at most one receivedprocessed picture of each of the PROTVS; and outputting plural pictures,wherein each output picture comprises of information corresponding tothe processed pictures in a respective DSOOOMAPP.
 2. The method of claim1, wherein a second plurality of the processed pictures of the SROTVS isreceived in the video stream after at least a portion of a secondplurality of the processed pictures of the FROTVS, wherein the firstreceived picture of the second plurality of the processed pictures ofthe SROTVS is received after the last received picture from the firstplurality of the processed pictures of the SROTVS.
 3. The method ofclaim 2, wherein the order of receiving the processed pictures of theFROTVS and SROTVS in the video stream is in accordance with eachsuccessively received picture from the FROTVS representing respectivelythe same picture of the video signal as each successively receivedpicture from the SROTVS.
 4. The method of claim 3, wherein successivelypictures of the second plurality of the processed pictures of the FROTVSare received in the video stream while successively receiving picturesof the first plurality of the processed pictures of the SROTVS.
 5. Themethod of claim 4, further comprising: receiving auxiliary informationthat identifies associated processed pictures of the PROTVS, wherein thedetermining of plural sets of one or more associated processed picturesof the PROTVS is in accordance to the received auxiliary information. 6.The method of claim 2, wherein each of the PROTVS corresponds to arespective decimated version of the video signal, and wherein eachreceived processed picture of each of the PROTVS corresponds to arespective decimated picture of the video signal.
 7. The method of claim6, wherein the first received picture from the first plurality of theprocessed pictures of the SROTVS is received in the video stream afterthe last received picture from the first plurality of the processedpictures of the FROTVS.
 8. The method of claim 7, wherein the firstreceived picture from the first plurality of the processed pictures ofthe SROTVS is received in the video stream while receiving the firstpicture from the second plurality of the processed pictures of theFROTVS.
 9. The method of claim 2, wherein the first received picturefrom the first plurality of the processed pictures of the SROTVS isreceived in the video stream prior to the first received picture fromthe second plurality of the processed pictures of the FROTVS.
 10. Themethod of claim 9, wherein the first received picture from the firstplurality of the processed pictures of the SROTVS is received in thevideo stream while receiving a picture from the first plurality of theprocessed pictures of the FROTVS.
 11. The method of claim 2, whereineach respective DSOOOMAPP includes one received processed picture ofeach of the PROTVS, and wherein each output picture is assembled frominformation corresponding to each of the processed pictures in therespective DSOOOMAPP.
 12. A method for receiving and outputting pluralrepresentations of a video signal, the method comprising: receivingplural representations of a video signal, wherein each of the pluralrepresentations of the video signal (PROTVS) includes successivecompressed pictures, wherein each of the PROTVS is associated with arespective pixel sampling lattice used to decimate each picture of thevideo signal, wherein each compressed picture of each of the PROTVSrepresents a respectively corresponding decimated picture of the videosignal; wherein each respective picture of the video signal isrepresented by at most one compressed picture of each of the PROTVS;determining respective sets of one or more compressed pictures of thePROTVS that represent each picture of the video signal; decompressingthe successive compressed pictures of each of the PROTVS; and outputtingplural pictures, wherein each outputted picture includes visualinformation corresponding to the respective decompressed version of eachof plural received compressed pictures that represent the samerespective picture of the video signal.
 13. The method of claim 12,further comprising: receiving auxiliary information that identify therespective sets of one or more compressed pictures of the PROTVS thatrepresent each picture of the video signal.
 14. The method of claim 12,wherein the PROTVS are received in respective video streams.
 15. Themethod of claim 12, wherein the PROTVS are received in a single videostream.
 16. The method of claim 12, further comprising: receiving afirst compressed picture representing a first picture of the videosignal; and delaying the outputting of a picture including visualinformation corresponding to the decompressed version of the firstcompressed picture until receiving and decompressing a second compressedpicture that representing the first picture of the video signal, whereinthe second compressed picture is received after the first compressedpicture.
 17. An apparatus for receiving and outputting pluralrepresentations of a video signal in a video stream, the apparatuscomprising: a processor; a processor-readable medium including one ormore instructions for: receiving a video stream, wherein the videostream includes plural representations of a video signal, wherein eachof the plural representations of the video signal (PROTVS) includes arespective sequence of compressed pictures, wherein each of the PROTVSis associated with a respective pixel sampling lattice used to decimateeach picture of the video signal, wherein each compressed picture ofeach of the PROTVS represents a respectively corresponding decimatedpicture of the video signal; determining plural sets of one or moreassociated compressed pictures of the PROTVS, wherein each compressedpicture in each determined set of one or more associated compressedpictures (DSOOOMACP) represents the same respective picture of the videosignal, wherein each respective DSOOOMACP includes at most one receivedcompressed picture of each of the PROTVS; and outputting pluralpictures, wherein each output picture comprises of informationcorresponding to the respective decompressed version of one or morereceived compressed pictures that represent the same respective pictureof the video signal.
 18. The system of claim 17, wherein each of theassociated compressed pictures in a first DSOOOMACP is received over arespective interval, wherein the start of a second interval including asecond associated compressed picture in the first DSOOOMACP is after theend of a first interval including a first associated compressed picturesin the first DSOOOMACP.
 19. The system of claim 17, wherein a firstplurality of compressed pictures is received over a first interval,wherein each of the first plurality of compressed pictures received overthe first interval corresponds respectively to a different DSOOOMACP.20. The system of claim 19, wherein a first DSOOOMACP includes a firstcompressed picture of the first plurality of compressed picturesreceived over a first interval, wherein a second compressed pictureincluded in the first DSOOOMACP is received over a third interval,wherein the start of the third interval is after the end of a secondinterval, wherein the start of the second interval is after the end ofthe first interval.